1. Introduction and Goals

This project analyses the LendingClub's loan dataset. The dataset includes loan application data, the information on granted loans and their status.

Goals

Build models for:

1. Classifying Accepted and Rejected loans
2. Estimating loan default risk (loan_status)
   • Can be used by potential investors into the platform to select a sample of loans to invest in (instead of provided grades/subgrade/etc)
3. Classifying loan grade and sub_grade

Classifying loan grade and sub_grade

The goal of this project is build several ML models which can be used to predict:

• Binary loan_status prediction model. We have used XGBoost and CatBoost to build a model which can be used to predict how likely an applicant is to default on a loan based on their information and the loan size, type and other specifics. This model can be used to decide whether the application should be approved or used by a third party investor to estimate risk. The goal of this model is to maximize the quality of probability prediction rather than the classification accuracy itself. In addition to estimating the default likelihood we'll also use this model to estimate:
  • Loan grades (A,B,C etc.) based on probability threshold, this does not directly attempt to predict the grades defined in the dataset but rather can be used as an alternative method to classify loan risk
  • Using this model we'll calculate the profitability and risk of a loan portfolio if only and provide an improved (more restrictive) method of evaluating loan applicants
• loan_grade prediction models, we'll evaluate and compare two different approaches
  • Multi classification model using XGBoost and CatBoost
  • Regression model that attempts to predict grade converted to a normalized numerical variable (predictions are rounded). While using a regression model for a classification problem is not ideal because it ignores the fact that spacing between different classes of ordinal features might differ (e.g. we don't know if the distance between grade A and B is the same as between B and C etc.)

Practical Applications

The default risk model can be used by potential investors to figure out which loans offered by the Lenders Club to invest in.

• e.g. a non experienced individual would likely base their decision by just looking into the grade, interest rate and maybe a few other variables and make their decision based on this.
• Our default risk model allows them to build an investment strategy based on their risk profile. Basically our goal is to provide a self-contained model that can be used (instead of just investing based on the grade/interest rate) to maximize returns based on acceptable risk.

Dropping cols where: loan_status is missing 2034631 -> 2034601

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

1.1 EDA

This is the exploratory analysis of the dataset and includes all the features shown in the (Features) notebook.

The EDA was done in parallel with model design process to minimize the influence analyzing manually analyzing the interaction between features might have had on our model design process.

Data selection

We have three target variables which we'll try to predict using in our following sections. They are shown here as:

• target_loan_status: whether the loan applicant has defaulted on their loan (0 or 1)
• pass_grade: the grade assigned to the loan by the "Lenders Club" (A,B,C,D etc.)
• pass_interest_rate: the interest rate set for the loan (converted to float)

['target__loan_status']

1.1.1 Correlation Matrix

Loan Status has strong correlation with:

• Applicant FICO (credit) score
• Term of the loan (longer loans are somewhat more to be defaulted on)
• While the correlation with the zip_code is almost 0.4 it is not significant (other variables like grade, sub_grade and interest rate are not included in our model because they are calculated based on estimated loan risk, we could still include them if our goal is to verify the quality of "Lenders Club" estimates we chose not to include them to build a "clean" alternative model using the same source data).

Loan Grade and Interest Rate have significant correlations with a much larger number of variables:

• Loan Term
• Loan amount and installment size
• FICO score
• Loan purpose
• Applicant income
• Income verification status etc.

This would indicate that the Lenders Club is trying to use a large number of variables to determine the grade and interest rate but that those variables are not necessarily as useful at predicting the likelihood of default.

Because the datatypes of features vary we had to use different methods to measure the strength and significance of each pair:

- Chi-Squared Test: Assesses independence between two categorical variables.  For bool-bool pairs due to categorical nature.

- Point Biserial Correlation: Measures correlation between a binary and a continuous variable. For bool-numerical pairs to account for mixed data types.

- Spearman's Rank Correlation: Assesses monotonic relationship between two continuous variables. Used for numerical-numerical pairs (for non-normally distributed data).

Since the Chi-Squared test outputs an unbound statistic/value which can't be directly compared to  pointbiserialr or Spearman Rank we have converted them to a  ` Cramér's V:` value which is normalized between 0 and 1. This was done to make the values in the matrix more uniform however we must note that Cramér's V and Spearman's correlation coefficients are fundamentally different statistics and generally can't be directly compared.

1.1.1 Relationships between individual variables

This part includes various charts showing the relationships between specific manually selected variables:

We can see that much fewer grade A and B loans were defaulted on while at C or below the likelihood of default becomes higher than average:

'Default Rate and Expected Return by Grade:'

	default_rate	mean_interest_rate	expected_return
pass__grade
A	0.06	7.11	6.68
B	0.13	10.68	9.25
C	0.22	14.02	10.87
D	0.30	17.72	12.34
E	0.38	21.14	13.02
F	0.45	24.92	13.68
G	0.50	27.72	13.82

The expected_return show the rate of return adjusted by the default rate. e.g. while almost 60% of all G grade loans are defaulted on you can still expect to make ~12% per year because the interest is very high. A grade loans tend to be relatively much less profitable despite the much lower risk.

Text(0.5, 1.025, 'Grade and Interest Rate')

There is quite a bit of overlap between interest rates for lower grades (D+).

Text(0.5, 1.025, 'Grade and Loan Amount')

Text(0.5, 1.025, 'Grade and Annual Income')

As we would expect the applications by individuals who tend to have very low incomes tend to receive lower grades. However, the rate of E grade loans increases significantly along with grade A > ~100k yearly income which is quite interesting and requires additional analysis because this does not apply to loans of other grades (we can see in charts below that the income of ~25% applicants is > 90k so this can't really be explained by random variation due to small sample size)

Text(0.5, 1.025, 'Grade and Fico Score')

Text(0.5, 1.025, 'Default Rate and Interest Rate')

Text(0.5, 1.025, 'Default Rate and Annual Income')

While low income individuals are much more likely to default the risk of default doesen't seems to decrease much or at as income goes up.

Text(0.5, 1.025, 'Loan Amount and Purpose')

count    1.210204e+06
mean     1.442061e+04
std      8.717951e+03
min      5.000000e+02
25%      8.000000e+03
50%      1.200000e+04
75%      2.000000e+04
max      4.000000e+04
Name: pass__loan_amnt, dtype: float64

Relat

'term_parser__term_binned vs pass__int_rate'

None

'target__loan_status_binned vs pass__int_rate'

None

'pass__loan_amnt_binned vs pass__int_rate'

None

'pass__annual_inc_binned vs pass__int_rate'

None

'pass__fico_range_high_binned vs pass__int_rate'

None

'pass__last_fico_range_high_binned vs pass__int_rate'

None

'pass__grade vs pass__int_rate'

None

'pass__home_ownership vs pass__int_rate'

None

'pass__purpose vs pass__int_rate'

None

'term_parser__term_binned vs pass__annual_inc'

None

'target__loan_status_binned vs pass__annual_inc'

None

'pass__loan_amnt_binned vs pass__annual_inc'

None

'pass__int_rate_binned vs pass__annual_inc'

None

'pass__fico_range_high_binned vs pass__annual_inc'

None

'pass__last_fico_range_high_binned vs pass__annual_inc'

None

'pass__grade vs pass__annual_inc'

None

'pass__home_ownership vs pass__annual_inc'

None

'pass__purpose vs pass__annual_inc'

None

'term_parser__term_binned vs pass__fico_range_high'

None

'target__loan_status_binned vs pass__fico_range_high'

None

'pass__loan_amnt_binned vs pass__fico_range_high'

None

'pass__int_rate_binned vs pass__fico_range_high'

None

'pass__annual_inc_binned vs pass__fico_range_high'

None

'pass__last_fico_range_high_binned vs pass__fico_range_high'

None

'pass__grade vs pass__fico_range_high'

None

'pass__home_ownership vs pass__fico_range_high'

None

'pass__purpose vs pass__fico_range_high'

None

1273506    689.0
324024     784.0
2066630    744.0
1975547    699.0
302723     684.0
           ...  
407        714.0
1247320    739.0
1824148    719.0
891280     734.0
582591     694.0
Name: pass__fico_range_high, Length: 1210204, dtype: float64

1.2.1 Variable Selection

1.2.2 Population Stability Index

We can use PSI to drop unstable features:

Interpreting PSI Values PSI = 0: No change in the distribution between the two datasets. 0 < PSI < 0.1: Indicates a minor change in distribution. Generally acceptable for most applications. 0.1 ≤ PSI < 0.25: Suggests a medium change, warranting investigation. PSI ≥ 0.25: Represents a significant change in the feature distribution, suggesting that the feature's behavior has altered considerably from the baseline. Features with high PSI are often considered unstable and may be dropped from the model to maintain its performance.

1210204

DropHighPSIFeatures(split_frac=0.6)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

{'term_parser__term': 0.0,
 'emp_length_parser__emp_length': 0.0003741361314581748,
 'target__loan_status': 0.0,
 'pass__loan_amnt': 0.003920832633671107,
 'pass__int_rate': 0.0980411585055524,
 'pass__installment': 0.0034312458504524098,
 'pass__annual_inc': 0.0004628902891250139,
 'pass__dti': 0.00860960003097421,
 'pass__delinq_2yrs': 3.3214883865642225e-09,
 'pass__fico_range_low': 0.0001729071232601537,
 'pass__fico_range_high': 0.0001729071232601537,
 'pass__inq_last_6mths': 0.00012248006077575135,
 'pass__open_acc': 0.001492233050605375,
 'pass__pub_rec': 0.0001235793133980767,
 'pass__revol_bal': 0.004023995601426297,
 'pass__revol_util': 0.0015717320542180126,
 'pass__total_acc': 0.001815363644818449,
 'pass__last_fico_range_high': 0.002840491100855753,
 'pass__last_fico_range_low': 0.002840491100855753,
 'pass__collections_12_mths_ex_med': 0.0,
 'pass__inq_fi': 0.0004968674485310809,
 'pass__inq_last_12m': 0.00019709687701067191,
 'pass__chargeoff_within_12_mths': 0.0,
 'pass__mort_acc': 0.000995348626684566,
 'pass__pub_rec_bankruptcies': 3.7430455156439416e-06,
 'pass__tax_liens': 0.0,
 'pass__tot_hi_cred_lim': 0.00011267406754624003,
 'pass__total_bal_ex_mort': 0.000485304089260657}

2. Model Design

As we've discussed at the begining our goal is to build ML models to predict these target features:

• Loan Default Risk (loan_status)
• Loan Grade (loan_grade)
• Loan Grade (loan_sub_grade)
• Interest Rate (int_rate)

In all cases we've either used XGBoost or CatBoost models (based on performance and training time).

2.1. Model Pipelines

We've used sci-kit lear pipelines to build and design our models. The data passed to the pipeline is preprocessed separately in src/preproc which filters only completed loans (based on loan_status) and performs other clean up and preprocessing (without fundamentally transforming or scaling etc.) to make all the features compatible with our ML models.

Example of a pipeline:

Pipeline(steps=[('preprocessing',
                 FunctionTransformer(func=<function convert_to_category at 0x7fc1b16239a0>)),
                ('feat_trans_inst_income_ratio', InstallementIncomeRatio()),
                ('feat_trans_DropLowImportanceFeatures',
                 DropLowImportanceFeatures(option=<Options.OFF: 0>)),
                ('feat_trans_loan_purpose', LoanPurposeRiskTransformer()),
                ('remove_columns',
                 FunctionTransformer...
                               feature_types=None, gamma=0, grow_policy=None,
                               importance_type=None,
                               interaction_constraints=None, learning_rate=0.1,
                               max_bin=None, max_cat_threshold=None,
                               max_cat_to_onehot=None, max_delta_step=None,
                               max_depth=7, max_leaves=None, min_child_weight=3,
                               missing=nan, monotone_constraints=None,
                               multi_strategy=None, n_estimators=50,
                               n_jobs=None, num_parallel_tree=None,
                               random_state=None, ...))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Each pipeline consists of these steps:

• a preprocessing function which handles data encoding if necessary (we do not encode categorical variables because XGBoost and CatBoost support them directly).
• a set of feature transformers selected based on hyper tuning results (the transformers were designed iteratively while designing the model and performing the EDA). The transformer merge/combine or otherwise modify various features based on subject knowledge (e.g. calculating various ratios or merging variables which represent basically the same features etc.)
• ML model.

2.2 Hyper-tuning

Because the dataset is very large we have split the tuning and model evaluation into separate notebooks, so they can be performed independently:

• Hyper tuning is first performed using default or fixed model hyperparameters to select the optimal transformers and their parameters.
• Hyperparameter tuning is performed using a fixed pipeline with transformers selected in the previous step.

This is a compromise approach, ideally we would use two nested loops with grid search performed on the outer level to select the transformers and a nested loop (using random or bayesian search) for model hyper tuning. However, considering the size of the dataset (and even if we use a relatively large sample) would be very slow because it would exponentially increase the number of required iterations.

The results are serialized to a binary file that includes the information on the model and training dataset.

2.3 Model Evaluation

Is performed by loading the hyper tuning information saved in the previous step, the model (with the same pipeline) is trained using the test training sample of the dataset, validated using CV and then evaluating using a separate training sample.

2.4 Production Deployment

The model from the previous step is serialized using (dill or joblib) and deployed to remote VM running a FastAPI backend that can be integrated into a website or other application.

2.1.1 Predicting Rejected Loans

Our first model is used to predict whether a loan application was accepted or rejected. It uses a table that was merged from the accepted and rejected loan (only the independent variables that are available in both are included).

Accepted/Rejected is encoded as 1/0 in our traing/test samples

2.1.1.1 Model Selection and Hypertuning

We've experimented using multiple models:

• XGBoost
• LGBM
• CatBoost

And multiple tuning target metrics:

• F1
• PR-AUC
• LogLoss

When predicting rejected application we have achieved the best performance when using an XGBoost (as fast or faster than CatBoost and LGBM with slightly better performance) tuning to maximize F1.

see the appendices for more details.

2.1.1.2 Training and Evaluating the Accepted/Rejected Application Model

The dataset is pre-split into a 20% test sample and a 80% training sample.

Overall our goal was to maximize the macro F1 (we have no reasons to priortize the accuracy for either class because the model doesn't really have any practical applications besides estimating the algorithm Lenders Club is using to reject applications and any potential biases that it might contain).

Additionally, we've also used PR-AUC to evaluate our model, it quantifies a classifier's ability to distinguish between classes under varying thresholds (however this will be more relevant for the default risk estimation model).

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

Training: XGBoostTuneF1 with: {'feat_trans_dummytransformer__option': 1, 'model__scale_pos_weight': 5, 'model__n_estimators': 150, 'model__min_child_weight': 2, 'model__max_depth': 7, 'model__learning_rate': 1, 'model__gamma': 0.1}
BaselineOnlyEmplDuration: 22.7 seconds
Training: XGBoostTuneF1 with: {'feat_trans_dummytransformer__option': 0, 'model__scale_pos_weight': 2, 'model__n_estimators': 200, 'model__min_child_weight': 3, 'model__max_depth': 4, 'model__learning_rate': 0.5, 'model__gamma': 0.05}
XGBoostTuneF1Rejected: 91.0 seconds

The models were trained using 5-Fold CV validation. We've evaluated them using a 20% test split (considering the size of the dataset using a much smaller test split would probably be acceptable):

The figure layout has changed to tight

We have been able to achieve very solid results both when tuning for F1 and PR-AUC with F1=~0.94 and PR-AUC =~0.98.

The model performs significantly better at predicting rejected loans (~17% of accepted loans are misclassified). Depending on how the model would be used we might consider using FBeta, non-macro F1 or even accuracy as our tuning target to maximize the performance for a specific class.

We can see that our model is signficantly better at predicting positive (i.e. loan accepted) than negative cases, with quite a few rejected cases being misclassified as accepted. We could potentially use different hyper-parameters to optimize for precision instead of overal predictive power if our business problem required that (we'll see some example of that when building the Default Risk model in part next section (2.2)

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

2.1.1.3 Conclusion

The model performance achieved using XGBoost is statisfactory (we've tried LGBM and Catboost and the performance was comparable). We've found that the Lending Club mainly bases their decision of whether to accept or decline an application based on the number years the application was employed:

• Almost all the applications Employment Length <= 1 are rejected (which is the majority of the rejected loans). So a dummy baseline model that only uses this variable is able to achieve ~87% accuracy and 0.87 macro F1.
• Using additional features can explain the remainder of the rejections (mainly Debt-To-Income Ratio and Risk_Score and Amount Requested) to some extent. Adding them to our model improve the performance to 94%/0.94.
• If the algorithm/model used by the platform is fixed and only uses the same features even such high performance might be considered to be to poor. However, if we assume that their decision process is not 100% consistent and decisions can still be made on a case by case basis the performance is probably close to being as good as it can be.

2.2.2 Training and Testing the Default Risk Model

The dataset is pre-split into a 20% test sample and a 80% training sample (10% of which is used for tuning):

Dropping cols where: loan_status is missing 2034631 -> 2034601

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

Training: XGBoostTuneF1 with: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 0, 'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.1, 'model__gamma': 0.4}
XGBoostTuneF1: 201.0 seconds
Training: XGBoostTunePRAUC with: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 0, 'model__scale_pos_weight': 5, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 7, 'model__learning_rate': 0.1, 'model__gamma': 0}
XGBoostTunePRAUC: 199.2 seconds

2.2.3 Results and Evaluation

Predicting default risk (loan_status) seems to be significantly harder than rejected/accepted loans. However, we were still able to achieve relatively good results (F1=~0.85).

The figure layout has changed to tight

Training: XGBoostTunePRAUC with: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0.05} XGBoostTunePRAUC: 1.1 seconds

The classification performance of all the models using the default 50% threshold is comparable (XGBoostTuneF1DummyDropFICO was included as a baseline).

We have chosen the XGBoostTunePRAUC configuration as our "production model" since it performs the best at predicting accurate probabilities (i.e. lowest log_loss).

XGBoostPRAUCBase (which achieves similar classification performance) uses a minimally processed dataset while XGBoostTunePRAUC uses several feature transformers (that were selected based on their impact on performances as part of our tuning process):

• DropLowImportanceFeatures, dropping features with importance close to 0.0 simplifies the model and has no negative impact on performance
• InstallementIncomeRatio, calculates the ratio between the income of the application and the expected installment size
• LoanPurposeRiskTransformer, uses target encoding for the Loan Purpose column by calculating the default risk based on it.

We have tried multiple other feature transformers (see src/feature_engineering.py) but they had negative or no signficant positive impact on performance.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

Evaluating the Best Performing (XGBoostTunePRAUC) Model

Average Default Rate by Grade in the Dataset:

	Counts	Proportion	Loan_Status_Ratio
A	211381	0.175	0.060
B	353495	0.292	0.134
C	343327	0.284	0.224
D	180840	0.149	0.304
E	84224	0.070	0.384
F	28719	0.024	0.451
G	8218	0.007	0.502

Evaluating Financial Returns/Performance

We'll use the model we have built to narrow down the sample of loans that have a lower risk to default.

This would be more useful to potential investors rather than the Lenders Club itself, they can use it in addition to the grade and other data provided by the Lenders Club to select the loans which not to invest it in.

The table below shows the total returns of all the loans in the dataset. We have to note that this is highly simplified because it assumes that all the loans that were defaulted on had to be completely written off but realistically this is not really the case:

• all lenders are likely to make at least some payments before they default
• defaulted loans can still be sold to third part for a fraction of the owed amount In future version of our model we should take these factors into account.

	Total Loan Amount	Total Interest Paid	Expected Return %	Total Return %	Total Loss
0	17451.88m	1770.28m	13.645651	10.143752	3757.82m

We have selected only the loans which are unlikely to default using the default 50% threshold (note that the predicted probabilities don't directly indicate the likelihood of default):

	Actual	Hypothetical
Total Loan Amount	17451.8813M	2455.6286M
Total Interest Paid	1770.2756M	310.9426M
Total Return %	10.14%	12.66%
Default Rate	19.96%	2.29%
Total Loss	3757.8181M	57.4278M
Losses Avoided	None	3700.3903M
Interest Lost	None	-1459.3330M
Total Applications Accepted	1210204	172025

Our model is fairly conservative and has rejected most of the loans offered by the Lenders Club (i.e. more than 80% are considered to be too risky). This would make it not very useful for the Lenders Club itself (however we might be able to improve this tuning the threshold). However, from the perspective of potential investors this is perfectly acceptable because they are likely to only invest into a small subset of available loans.

Using our model would allow investors to signficantly improve their returns by 3% or so, further improvements might be possible by improving the threshold (we'll look into this in the next section).

Calculating Probability Thresholds

If we're using our binary classification model to assign grades to loans based on default probabilities we can't directly try to predict the grades from the source dataset.

Basically what we can do is to use the source dataset as a baseline and try to determine whether our model can outperform the method the Lending Club is using for determine grades, we'll try to maintain the same ratio of grades while trying to minimize the default risk for loans included in top grades (basically we want to make sure that's the spread of default risk between grades is maximized i.e. default risk for grade A is as close to 0% and for grade F as close to 100% as possible while also ensuring that the risk increased with continuously with grade)

Attempt to set non-positive xlim on a log-scaled axis will be ignored.

The chart shows the performance of the if only individual with stroke Prob. > T are selected. Additionally the overlay indicates the number of people whose predicted P is in an given range. The overlays can be used to selected the most at risk individual based on the probability predicted for them

We have used the predicted probabilities to suggest alternative grades for each loan. Expected D.R is the default rate based on original grades. We can see our grades are much better at excluding higher risk loans from higher grades.

Generally this does not necessarily mean that source grades are inaccurate or objectively worse, the investors club is likely using a more straightforward and transparent algorithm for assigning them and investors should be using additional criteria when selecting which loans to invest anyway.

Comparing Interest Rates for Predicted Grades

In this section we'll try to evaluate investment strategies that are entirely based on loan grades (either the original ones or ones provided by our model)

Predicted Grade Real Return Rates

'Returns and Default Rate by Grade in Source Dataset:'

	default_rate	mean_interest_rate	expected_return
pass__grade_merged
A	0.06	7.11	6.68
B	0.14	10.69	9.23
C	0.22	14.03	10.87
D	0.30	17.73	12.37
E-F-G	0.41	22.47	13.35

'Returns and Default Rate Based on Default Probability Model:'

	default_rate	mean_interest_rate	expected_return
predicted_grades
A	0.003	10.519	10.485
B	0.009	12.168	12.063
C	0.083	13.951	12.796
D	0.558	14.732	6.507
E-F-G	0.902	16.846	1.652

We can see that while using our model to invested in the worst grades loans would lead to relatively poor returns investing in only in C grade or higher rated loans would provide better returns than using the Lenders Club's values to invest in the same loans:

• Investing only in C loans would provide almost the same returns as the far riskier E-F-G loans using the original grades.
• Only investing in A loans still yields ~10%, considering that the default rate is extremely low it might still be a preferable option to investing into high risk loans.

Basically using our model the investor would be able to minimize default risk while still getting decent returns i.e. if they only invested in loans with an estimated 10% default likelyhood or less using the original grades (so only A grade loans) they'd only make 6-7% while using our model this can be increased to more than 10%.

Limitations

We must take into account that this calculation is very simplified and therefore probably not accurate because, our model and calculations assume that Loss Given Default (LGD) is 100% and that Exposure at Default (EAD) is also 100% which is not accurate: - loans usually are not defaulted on immediately (i.e. some payments are almost always received) - most applicants who are unable to keep up with their payments are given some leeway and {PAYMENT PLAN} - failed loans are often sold to third parties for at least some fraction of the owed amount

Therefore, to provide a more accurate estimate and to figure out whether our model would actually be useful we'd need to incorporate the actual amounts received etc.

2.3.0 Estimating Grade and Sub-Grade

We've used a multi classification model to predict the grade and sub grade process. Unlike for the previous model only different configuration of XGBoost were used. Our tuning process was similar to that used by previous model:

• Selection of optimal feature transformers (using grid search).
• Hypertuning for different traget metrics (PR-AUC, logloss and macro F1) score to select the optimal hyperparameters.

See the appendices for more details.

Dropping cols where: grade is missing 2034631 -> 2034601

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

Total samples loaded : 2034601

2.3.1 Training and Evaluation

We've selected a configuration tuned for F1 as our "production" model. However, additional configurations were tested. It's worth noting that tuning for PR-AUC achieved comparable performance however the training time increase significantly, the configuration used max_depth = 12, learning_rate while= 0.2 while 7/0.3 and setting gamma to 0.1 (minimum reduction in loss required to make a further partition on a leaf node of the tree) instead of 0.0 was sufficient.

Training: XGBoostF1Multiclass with: {'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.3, 'model__gamma': 0.1}
XGBoostF1Multiclass: 3934.5 seconds

Pipeline(steps=[('preprocessing',
                 FunctionTransformer(func=<function convert_to_category at 0x7fe73d247010>)),
                ('feat_trans_loan_grade', LoanGradeTransformer()),
                ('feat_trans_dti_inc_joint', JointApplicationTransformer()),
                ('feat_trans_fico_score', FICOScoreTransformer()),
                ('feat_trans_delinquency', DelinquencyTransformer()),
                ('feat_trans_inst_income_ratio', Ins...
                               feature_types=None, gamma=0.1, grow_policy=None,
                               importance_type=None,
                               interaction_constraints=None, learning_rate=0.3,
                               max_bin=None, max_cat_threshold=None,
                               max_cat_to_onehot=None, max_delta_step=None,
                               max_depth=7, max_leaves=None,
                               min_child_weight=2.5, missing=nan,
                               monotone_constraints=None, multi_strategy=None,
                               n_estimators=150, n_jobs=None,
                               num_parallel_tree=None,
                               objective='multi:softprob', ...))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Results:

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

	Feature	Importance
0	term_parser__term	0.283267
1	emp_length_parser__emp_length	0.001460
2	zip__zip_code	0.015377
3	pass__loan_amnt	0.114238
4	pass__installment	0.083785
5	pass__home_ownership	0.001546
6	pass__annual_inc	0.017439
7	pass__verification_status	0.030414
8	pass__purpose	0.028362
9	pass__addr_state	0.002459
10	pass__dti	0.017551
11	pass__delinq_2yrs	0.001861
12	pass__fico_range_low	0.133956
13	pass__fico_range_high	0.000000
14	pass__inq_last_6mths	0.034767
15	pass__mths_since_last_delinq	0.001965
16	pass__mths_since_last_record	0.002540
17	pass__open_acc	0.002886
18	pass__pub_rec	0.003241
19	pass__revol_bal	0.005628
20	pass__revol_util	0.007873
21	pass__total_acc	0.007038
22	pass__initial_list_status	0.061059
23	pass__last_fico_range_high	0.040828
24	pass__last_fico_range_low	0.000000
25	pass__collections_12_mths_ex_med	0.001278
26	pass__mths_since_last_major_derog	0.001657
27	pass__application_type	0.004819
28	pass__annual_inc_joint	0.003288
29	pass__dti_joint	0.005941
30	pass__verification_status_joint	0.006741
31	pass__inq_fi	0.021314
32	pass__inq_last_12m	0.019824
33	pass__chargeoff_within_12_mths	0.001644
34	pass__mort_acc	0.009777
35	pass__pub_rec_bankruptcies	0.005268
36	pass__tax_liens	0.003532
37	pass__tot_hi_cred_lim	0.012653
38	pass__total_bal_ex_mort	0.002723

While the performance for A,B,C grades is relatively acceptable (F1 > ~0.8) performance when predicting the lower grades is very poor (especially for grade G which is almost never classified correctly).

As we have learnt when building our default risk model there difference in returns and other features does not vary as much for lower quality grades which might make it hard to distinguish them. Therefore we'll use the same approach and merge E-F-G grades into a single group.

Merging E-F-G loan grades into a single group

Training: XGBoostF1Multiclass with: {'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.3, 'model__gamma': 0.1}
XGBoostF1Multiclass: 224.5 seconds

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

	Feature	Importance
0	term_parser__term	0.186359
1	emp_length_parser__emp_length	0.007387
2	zip__zip_code	0.061612
3	pass__loan_amnt	0.074794
4	pass__installment	0.059826
5	pass__home_ownership	0.008242
6	pass__annual_inc	0.018254
7	pass__verification_status	0.024264
8	pass__purpose	0.028074
9	pass__addr_state	0.012059
10	pass__dti	0.018394
11	pass__delinq_2yrs	0.009971
12	pass__fico_range_low	0.122972
13	pass__fico_range_high	0.000000
14	pass__inq_last_6mths	0.036808
15	pass__mths_since_last_delinq	0.008587
16	pass__mths_since_last_record	0.008891
17	pass__open_acc	0.008136
18	pass__pub_rec	0.008436
19	pass__revol_bal	0.010784
20	pass__revol_util	0.012640
21	pass__total_acc	0.010719
22	pass__initial_list_status	0.070690
23	pass__last_fico_range_high	0.031447
24	pass__last_fico_range_low	0.000000
25	pass__collections_12_mths_ex_med	0.007279
26	pass__mths_since_last_major_derog	0.008355
27	pass__application_type	0.014726
28	pass__annual_inc_joint	0.012731
29	pass__dti_joint	0.011981
30	pass__verification_status_joint	0.011583
31	pass__inq_fi	0.016773
32	pass__inq_last_12m	0.016616
33	pass__chargeoff_within_12_mths	0.006558
34	pass__mort_acc	0.009786
35	pass__pub_rec_bankruptcies	0.010853
36	pass__tax_liens	0.011060
37	pass__tot_hi_cred_lim	0.013631
38	pass__total_bal_ex_mort	0.008723

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[30], line 11
      8 model = cv_results_all_models[model_key].test_data.test_model.named_steps["model"]
     10 booster_model = model.get_booster()
---> 11 shap_values = booster_model.predict(
     12     xgboost.DMatrix(X_test, enable_categorical=True), pred_contribs=True
     13 )
     14 graph.render_shap_plot(shap_values, X_test, model_title=model_key)

File ~/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py:2298, in Booster.predict(self, data, output_margin, pred_leaf, pred_contribs, approx_contribs, pred_interactions, validate_features, training, iteration_range, strict_shape)
   2295 shape = ctypes.POINTER(c_bst_ulong)()
   2296 dims = c_bst_ulong()
   2297 _check_call(
-> 2298     _LIB.XGBoosterPredictFromDMatrix(
   2299         self.handle,
   2300         data.handle,
   2301         from_pystr_to_cstr(json.dumps(args)),
   2302         ctypes.byref(shape),
   2303         ctypes.byref(dims),
   2304         ctypes.byref(preds),
   2305     )
   2306 )
   2307 return _prediction_output(shape, dims, preds, False)

KeyboardInterrupt: 

Classifying Subgrades

Dropping cols where: sub_grade is missing 90000 -> 89996

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

Total samples loaded : 89996

Predicting Sub-grades

We've attempted to build a model which predicts individual sub-grades (e.g. A1, A2 ... G5) in addition to top level grades. An XGBoost multi-classificaiton model was used, however the performance was unsatisfactory. We've included the overal performance summary and feature importances below. However, we decide to not provide an indepth analysis because the model would not be useful for any practical applications (a different approach would probably suit this problem better due to the high number of classes and their ordinal nature)

Conclusion

• As could have been expected the multi classification model performs significantly worse than a binary classifier.

• Performance when predicting the lower grades, especially E-F-G is poor. This might indicate that:

  • The algorithm used for setting the grades is not entirely fixed and the provided features can't be consistently. Classes E, F, and G might have features that are not as distinct from each other as those separating A, B, and C. This overlap can confuse the model, especially if the ordinal relationship leads to gradual changes in feature values across classes.
  • The features used might not effectively capture the variance within E, F, G, or the ordinal relationship between classes. Features that are good at differentiating A, B, C might not scale well into distinguishing E, F, G.

• The model complexity might not be sufficient to capture the nuances that would differentiate between E, F, and G. This could be due to too few trees, shallow trees, or not enough iterations. e.g.: If XGBoost is tuned with a low depth parameter, it might capture the broad differences between A, B, C well but fail to model the subtleties needed for E, F, G. However we have trained a configuration with a more complex model and the performance was not signficantly different. This might allow use to reject this hypothesis but additional testing and analysis might be necessary.

Dropping cols where: int_rate is missing 2034631 -> 2034601

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

Training: XGBoostRegressorTuneRMSE with: {'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'model__n_estimators': 50, 'model__max_depth': 4, 'model__learning_rate': 0.7, 'model__gamma': 0.1}
XGBoostIntRateRegressorMSE: 222.2 seconds

The deviations at the extremes indicate thar our data has heavier tails than a normal distribution. This means:

• Near the center (-2 to 2): The model's predictions are generally accurate, with errors distributed in a manner expected by random chance alone.
• Below -2 and above 2: The presence of outliers or extreme values our model didn't predict accurately, suggesting that for very low or very high values, the model's predictions become less reliable.

Conclusions

Limitations and Future Improvements

• We've applied relatively few feature engineering techniques in our project. While the rejected loans dataset is fairly simple trying to do that when using a complex model like XGBoost might not be necessary however the features in the accepted loans dataset are significantly more complex.

  • In addition to our feature transformers and using model importancess/SHAP plots to eliminate features we should use additional techniques (we've used PSI to eliminate unstable feature however that did not improve the performance so this might not be necessary with the dataset we have)

  • Clustering, might help us to identify inherent groupings or segments in the data which the model might be missing on its own. Clusters could can be used as new features, potentially capturing non-linear relationships that are not explicitly modeled in the original feature set.

  • Featuretools and other automated feature engineering packages.

• Tuning process can be improved by using bayesian tuning to speed it up (we've used Optuna initially)

• Model sizes, some of the models are very large, we should look into:

  • simplifying the XGBoost models themselves (by including less features and tweaking other parameters as long as that does not decrease performance)
  • using more efficient structures for serialization, we should consider optimizing the dataclasses used for storing the model and related metadata and consider using a prebuilt solution or a library instead of building our own custom methods.

Outcomes

Overal the performance of the rejected/accepted and loan default risk classifier model is acceptable however the multi classification model for predicting grades under-perform to a significant degree (however there unlikely to be any practical application for such models)

Appendices

Individual feature distributions and other supplementary data

4.1 Rejected/Accepted Loans Dataset

This is a supplement to the main EDA notebook (1.1 EDA) that includes the individual analysis of all numerical and categorical variables. The insights obtained here together with the analysis of interaction between individual variables was used to build the preprocessing pipeline and individual transformers (for feature engineering) that were used when building our models (see the EDA notebook for additional details).

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

'Total samples in dataset: 732548'

'Features with NaN Values:'

	Total NaN Values	Proportion NaN (%)
emp_length_parser__emp_length	32965	5.0
pass__loan_amnt	4	0.0
pass__last_fico_range_high	6979	1.0
pass__dti	374	0.0

'Total Duplicate Rows: 1.4158799150362844'

	emp_length_parser__emp_length	target__Accepted	zip__zip_code	pass__loan_amnt	pass__addr_state	pass__last_fico_range_high	pass__dti
3311572	0.0	0	895	1000.0	NV	0.0	0.00
3212643	0.0	0	402	2000.0	KY	0.0	0.00
2529965	0.0	0	290	1000.0	SC	0.0	0.00
2408637	0.0	0	061	1000.0	CT	0.0	0.00
2670565	0.0	0	235	1200.0	VA	0.0	0.00
...	...	...	...	...	...	...	...
3446532	0.0	0	338	2000.0	FL	0.0	0.00
3275040	0.0	0	100	1000.0	NY	0.0	0.00
2745815	0.0	0	114	20000.0	NY	NaN	-0.01
2879190	0.0	0	010	1500.0	MA	630.0	0.00
3295231	0.0	0	344	10000.0	FL	0.0	0.00

10372 rows × 7 columns

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

Text(0.5, 1.025, 'Application Status Based on Employment Duration')

4.2 Accepted Loans Dataset

This is a supplement to the main EDA notebook (1.1 EDA) that includes the individual analysis of all numerical and categorical variables. The insights obtained here together with the analysis of interaction between individual variables was used to build the preprocessing pipeline and individual transformers (for feature engineering) that were used when building our models (see the EDA notebook for additional details).

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

	term_parser__term	emp_length_parser__emp_length	target__grade	pass__loan_amnt	pass__int_rate	pass__installment	pass__sub_grade	pass__home_ownership	pass__annual_inc	pass__verification_status	pass__purpose	pass__zip_code	pass__addr_state	pass__dti	pass__delinq_2yrs	pass__fico_range_low	pass__fico_range_high	pass__inq_last_6mths	pass__mths_since_last_delinq	pass__mths_since_last_record	pass__open_acc	pass__pub_rec	pass__revol_bal	pass__revol_util	pass__total_acc	pass__initial_list_status	pass__last_fico_range_high	pass__last_fico_range_low	pass__collections_12_mths_ex_med	pass__mths_since_last_major_derog	pass__application_type	pass__annual_inc_joint	pass__dti_joint	pass__verification_status_joint	pass__inq_fi	pass__inq_last_12m	pass__chargeoff_within_12_mths	pass__mort_acc	pass__pub_rec_bankruptcies	pass__tax_liens	pass__tot_hi_cred_lim	pass__total_bal_ex_mort	pass__loan_status
2086299	36	1.0	0	14000.0	6.08	426.42	A2	MORTGAGE	70000.0	Not Verified	debt_consolidation	176xx	PA	31.10	0.0	800.0	804.0	0.0	NaN	NaN	18.0	0.0	22237.0	32.3	33.0	w	824.0	820.0	0.0	NaN	Joint App	350000.0	8.61	Not Verified	1.0	0.0	0.0	1.0	0.0	0.0	492126.0	78093.0	Current
1233152	60	NaN	4	35000.0	19.52	917.97	E2	MORTGAGE	120000.0	Verified	debt_consolidation	799xx	TX	23.75	1.0	710.0	714.0	3.0	22.0	NaN	11.0	0.0	49315.0	43.7	34.0	f	804.0	800.0	0.0	NaN	Individual	NaN	NaN	None	NaN	NaN	0.0	3.0	0.0	0.0	390414.0	90585.0	Fully Paid
461729	36	10.0	0	10000.0	7.34	310.33	A4	MORTGAGE	70000.0	Not Verified	debt_consolidation	063xx	CT	14.72	0.0	685.0	689.0	0.0	NaN	106.0	6.0	1.0	21943.0	47.5	16.0	w	749.0	745.0	0.0	NaN	Individual	NaN	NaN	None	0.0	0.0	0.0	1.0	1.0	0.0	56200.0	26022.0	Fully Paid
738295	60	10.0	3	21600.0	17.27	539.96	D2	MORTGAGE	96000.0	Verified	debt_consolidation	482xx	MI	15.83	1.0	690.0	694.0	1.0	15.0	NaN	11.0	0.0	14751.0	71.3	32.0	w	639.0	635.0	0.0	42.0	Individual	NaN	NaN	None	3.0	1.0	0.0	3.0	0.0	0.0	242183.0	76547.0	Fully Paid
864585	36	3.0	1	3300.0	10.07	106.60	B2	RENT	22000.0	Verified	other	490xx	MI	2.73	0.0	730.0	734.0	0.0	NaN	NaN	2.0	0.0	680.0	42.5	3.0	w	764.0	760.0	0.0	NaN	Individual	NaN	NaN	None	0.0	0.0	0.0	0.0	0.0	0.0	1600.0	680.0	Current

'Features with NaN Values:'

	Total NaN Values	Proportion NaN (%)
emp_length_parser__emp_length	1325	15.0
pass__dti	2	0.0
pass__mths_since_last_delinq	4587	51.0
pass__mths_since_last_record	7593	84.0
pass__revol_util	4	0.0
pass__mths_since_last_major_derog	6671	74.0
pass__annual_inc_joint	8545	95.0
pass__dti_joint	8545	95.0
pass__verification_status_joint	8556	95.0
pass__inq_fi	3422	38.0
pass__inq_last_12m	3422	38.0
pass__mort_acc	179	2.0
pass__pub_rec_bankruptcies	7	0.0
pass__tot_hi_cred_lim	275	3.0
pass__total_bal_ex_mort	179	2.0

	Total NaN Values	Proportion NaN (%)
pass__verification_status_joint	8556	95.0

count              444
unique               3
top       Not Verified
freq               212
Name: pass__verification_status_joint, dtype: object

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

2.1 Predicting Rejected/Accepted Loans

The model uses a dataset merged which combines accepted_2007_to_2018q4.csv and rejected_2007_to_2018Q4.csv (all feature the columns shared by both datasets are included).

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

We have excluded all of the rows in the rejected applications dataset after 2013-11-05 because that applications after that data no longer used a standard FICO score which might make it hard to directly compare to older applications or accepted applications (which only contain a FICO score). The target values are in the target__Accepted columns (0=rejected, 1=accepted):

	Total Rows	Proportion
1	452597	0.62
0	279951	0.38

emp_length_parser__emp_length     float64
target__Accepted                    int64
zip__zip_code                    category
pass__loan_amnt                   float64
pass__addr_state                 category
pass__last_fico_range_high        float64
pass__dti                         float64
dtype: object

2.1. Tuning the Binary Default Risk Classifier

We'll tune multiple XGBoost model which predict the target__Accepted column using F1 and PR-AUC (we have tried other metrics however they generally provide the same or inferior results)

Tunning: - transformers: transformers: 1
 total options: 1
[<class 'shared.ml_config_core.AcceptedRejectedDropAllButEmployment'>]
search_grid:
{'feat_trans_dummytransformer__option': [1]}

 - model: BaselineOnlyEmplDuration n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4], 'model__scale_pos_weight': [1, 2, 3, 5, 10]}
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Best Transformer Parameters: {'feat_trans_dummytransformer__option': 1}
Fitting 5 folds for each of 70 candidates, totalling 350 fits
Best Model Parameters: {'model__scale_pos_weight': 5, 'model__n_estimators': 150, 'model__min_child_weight': 2, 'model__max_depth': 7, 'model__learning_rate': 1, 'model__gamma': 0.1}
BaselineOnlyEmplDuration Fit, total time:7.7

The best_score is the metric used for tuning by that specific pipeline (i.e. either F1 or PR-AUC so the values are not directly comparable):

	best_score	best_params
model_key
BaselineOnlyEmplDuration	0.903298	{'feat_trans_dummytransformer__option': 1, 'model__scale_pos_weight': 5, 'model__n_estimators': 150, 'model__min_child_weight': 2, 'model__max_depth': 7, 'model__learning_rate': 1, 'model__gamma': 0.1}

Parameter hypertuning results:

'BaselineOnlyEmplDuration:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
0	0.057162	0.003184	0.006582	0.000171	5	150	2.00	7	1.00	0.10	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1
31	0.061090	0.001375	0.006553	0.000190	5	150	1.00	4	0.30	0.10	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1
33	0.069988	0.001930	0.006398	0.000190	5	200	2.00	4	0.90	0.05	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1
68	0.071286	0.001654	0.006120	0.001055	3	200	0.10	12	0.40	0.30	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1
35	0.071829	0.001900	0.006756	0.000160	2	200	2.00	6	0.70	0.00	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1
36	0.037220	0.000559	0.006387	0.000111	2	50	0.50	6	0.05	0.10	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1
38	0.086523	0.001490	0.006706	0.000199	2	250	3.00	10	0.70	0.05	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1
39	0.089614	0.011778	0.007924	0.002534	2	250	2.00	6	0.50	0.10	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1
41	0.118928	0.006213	0.008928	0.001057	3	250	3.00	4	0.05	0.30	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1
42	0.093364	0.007321	0.006916	0.000270	5	250	0.25	12	0.30	0.40	0.904459	0.907937	0.904251	0.897959	0.901882	0.903298	0.003295	1

2.2 Predicting Default Risk

Our goal is to build a default risk estimation model that could be used by potential investors to build a strategy for selecting the optimal loans to invest in on the platform.

2.2.1 Tuning the Binary Default Risk Classifier

We'll tune multiple XGBoost model which predict the loan_status column using F1 and PR-AUC (we have tried other metrics but they gennerally provide the same or inferior results)

Dropping cols where: loan_status is missing 226070 -> 226067

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

Tuning Results

The table below shows the highest target score (either F1 or PR-AUC) achieved for each model configuration:

	best_score	best_params
model_key
XGBoostF1Base	0.769397	{'model__scale_pos_weight': 3, 'model__n_estimators': 50, 'model__min_child_weight': 2, 'model__max_depth': 10, 'model__learning_rate': 0.05, 'model__gamma': 0.05}
XGBoostTuneF1	0.773237	{'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 200, 'model__min_child_weight': 2, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostTuneF1DummyDropFICO	0.397436	{'feat_trans_dummy_drop_FICO__option': 1, 'model__scale_pos_weight': 5, 'model__n_estimators': 100, 'model__min_child_weight': 2, 'model__max_depth': 5, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostPRAUCBase	0.765507	{'model__scale_pos_weight': 10, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 12, 'model__learning_rate': 0.3, 'model__gamma': 0.3}
XGBoostTunePRAUC	0.829786	{'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0.05}

Preprocessing and Transformer Tuning

We have attempted to performance various feature transformations* to our dataset which could potentially improve performance (i.e. binning or merging certain variables etc.). Because we have a high number of different combinations and because invidual transformers might perform differently when tuning for certain metrics we've also included them in our tuning process included:

1. Grid search is performed with every combination of transformers using "semi-optimal" (based on previous tuning) hyperparameter defined for each specific model type.
2. Random search is used to select the optimal hyperparameter

• see src/feature_engineering/py for full list of transformers.

XGBoostF1Base and XGBoostPRAUCBaseuse a minimally processed (only clean up and some encoding without changing the structure) dataset.

XGBoostF1Base and XGBoostTuneF1 additionally use severeal feature transformers (that combine several features, calculate their ratios or use target encoding for specific categorical columns, see feature_engineering.py). Only the transformers that resulted in a signficant performance improvement are included.

XGBoostTuneF1DummyDropFICO is a baseline model which was trained after excluded the FICO score columns (which have the highest importance score for other models)

While we have tried LightGBM, CatBoost and other models initially (see shared/ml_config_core for various tested configurations) we've narrowed to these two configurations because they provided the best (or approximately the same as) results while also being the fastest to train.

The parameters tuned to maximize F1 generally provide the best performance for binary classification, however the PR-AUC configuration results in more accurate probabilities because (which might be preferable because our goal is to build a model to select the loans to invest in)

Results based on transformer combinations

0 = disabled, some transformers have multiple different parameters (e.g. 1,2..)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 for model_key, model_data in tuning_results.items():
      2     transformer_tun_all_civ_results = model_data.transformer_tun_all_civ_results
      3     if transformer_tun_all_civ_results is not None:

NameError: name 'tuning_results' is not defined

We have built a simple regression model to evaluate the impact of each transformer and the hyperparameter. This is just to give a rough estimate and to allow comparing individual parameters (the values themselves are not particularly meaningful)

'Transformer impact for XGBoostTuneF1'

	Transformer_Option	Impact
0	feat_trans_DropLowImportanceFeatures__option_1	0.0000
1	feat_trans_loan_purpose__option_1	-0.0014

'Transformer impact for XGBoostTuneF1'

	Transformer_Option	Impact
14	model__min_child_weight_2.5	0.0344
15	model__min_child_weight_3	0.0342
11	model__min_child_weight_1	0.0340
12	model__min_child_weight_1.5	0.0334
13	model__min_child_weight_2	0.0284
25	model__learning_rate_0.4	0.0204
22	model__learning_rate_0.1	0.0190
10	model__min_child_weight_0.75	0.0164
9	model__min_child_weight_0.5	0.0139
28	model__learning_rate_0.7	0.0137

'Transformer impact for XGBoostTunePRAUC'

	Transformer_Option	Impact
0	feat_trans_DropLowImportanceFeatures__option_1	0.0000
1	feat_trans_loan_purpose__option_1	-0.0006

'Transformer impact for XGBoostTunePRAUC'

	Transformer_Option	Impact
15	model__min_child_weight_3	0.0266
14	model__min_child_weight_2.5	0.0252
13	model__min_child_weight_2	0.0237
22	model__learning_rate_0.1	0.0189
12	model__min_child_weight_1.5	0.0180
11	model__min_child_weight_1	0.0176
23	model__learning_rate_0.2	0.0175
21	model__learning_rate_0.05	0.0151
4	model__n_estimators_100	0.0145
10	model__min_child_weight_0.75	0.0108

Parameter hypertuning results:

'XGBoostF1Base:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
3	2.539655	0.033946	0.029509	0.000972	5	50	2.50	12	0.05	0.30	0.782170	0.766917	0.781803	0.763786	0.766696	0.772274	0.008007	1
44	0.343730	0.011107	0.020059	0.000772	3	50	3.00	4	0.01	0.30	0.785016	0.773770	0.773289	0.759934	0.767932	0.771988	0.008202	2
66	3.856601	0.091281	0.057155	0.001304	3	200	2.00	7	0.01	0.00	0.777414	0.771641	0.770237	0.760291	0.770270	0.769971	0.005515	3
43	1.857459	0.005767	0.041977	0.004333	3	100	3.00	10	0.20	0.05	0.775111	0.776093	0.769784	0.759340	0.751838	0.766433	0.009415	4
4	2.330039	0.062270	0.039948	0.003183	2	250	2.00	12	0.20	0.10	0.783394	0.772851	0.771898	0.740873	0.756458	0.765094	0.014850	5
39	0.534821	0.018117	0.022498	0.000576	3	50	1.50	5	0.10	0.40	0.779496	0.755663	0.773508	0.758621	0.757119	0.764881	0.009720	6
6	3.377790	0.073363	0.039932	0.000557	2	150	0.75	12	0.10	0.40	0.771062	0.774908	0.776065	0.750896	0.743879	0.763362	0.013334	7
51	1.372830	0.036773	0.042140	0.001267	3	200	3.00	5	0.20	0.30	0.772887	0.779809	0.762923	0.752577	0.747529	0.763145	0.012068	8
20	3.694322	0.180248	0.063709	0.003112	3	200	1.00	7	0.05	0.40	0.770318	0.767748	0.775044	0.745234	0.752475	0.762164	0.011349	9
56	5.730208	0.081080	0.069219	0.002336	5	200	1.00	12	0.10	0.05	0.768812	0.774955	0.770209	0.743567	0.748593	0.761227	0.012635	10

'XGBoostTuneF1:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
7	0.963510	0.087027	0.033874	0.009908	2	50	2.5	7	0.10	0.40	0.791774	0.782684	0.779983	0.764706	0.759539	0.775737	0.011896	1
42	2.094696	0.097653	0.048334	0.006060	3	150	3.0	7	0.10	0.40	0.784722	0.770563	0.771105	0.760611	0.770740	0.771548	0.007681	2
53	1.193887	0.097053	0.041867	0.005184	3	200	2.5	4	0.10	0.00	0.778982	0.780811	0.766028	0.761364	0.764103	0.770258	0.008029	3
8	2.232088	0.126904	0.034808	0.001156	10	250	2.5	10	0.20	0.30	0.774138	0.774417	0.775758	0.759026	0.765018	0.769671	0.006550	4
32	1.556768	0.039411	0.033771	0.001642	3	250	2.5	12	0.40	0.05	0.783242	0.772686	0.774077	0.752389	0.751616	0.766802	0.012619	5
20	9.254182	0.111213	0.084302	0.002762	3	200	1.5	10	0.01	0.10	0.768028	0.766581	0.777391	0.756573	0.760181	0.765751	0.007164	6
17	1.689366	0.039251	0.043464	0.001895	5	150	3.0	10	0.40	0.00	0.774423	0.770723	0.777877	0.747598	0.756561	0.765436	0.011493	7
5	1.438166	0.083314	0.040884	0.004518	1	250	2.0	4	0.01	0.05	0.773709	0.779197	0.767377	0.764388	0.739336	0.764802	0.013725	8
51	4.991368	0.143159	0.075015	0.001778	1	250	2.5	10	0.05	0.05	0.778087	0.771084	0.765757	0.755677	0.751908	0.764503	0.009653	9
25	2.671416	0.080313	0.046523	0.002202	3	200	1.0	10	0.20	0.05	0.777174	0.776065	0.760436	0.756422	0.752328	0.764485	0.010240	10

'XGBoostTuneF1DummyDropFICO:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
60	0.482940	0.096194	0.025295	0.003457	3	50	2.5	4	0.10	0.30	0.378335	0.358429	0.367089	0.354430	0.348185	0.361293	0.010499	1
6	4.331436	0.090352	0.042518	0.001584	10	50	2.0	12	0.01	0.00	0.364597	0.355685	0.364871	0.346712	0.359761	0.358325	0.006724	2
50	4.146137	0.115205	0.095537	0.026142	3	250	2.0	6	0.01	0.10	0.377551	0.343830	0.354256	0.363636	0.348206	0.357496	0.012029	3
37	1.436562	0.053556	0.031436	0.001343	10	50	3.0	7	0.01	0.05	0.349868	0.349132	0.354515	0.356000	0.350133	0.351930	0.002777	4
33	16.386426	0.307811	0.166987	0.054258	10	200	0.5	12	0.01	0.00	0.343019	0.333096	0.316703	0.316948	0.346797	0.331313	0.012647	5
46	1.713501	0.065415	0.040049	0.002440	3	100	1.0	6	0.01	0.10	0.358156	0.310406	0.308113	0.320298	0.302412	0.319877	0.019993	6
13	0.523651	0.019858	0.024155	0.001896	3	50	2.0	5	0.40	0.30	0.323194	0.299065	0.302281	0.309160	0.306064	0.307953	0.008348	7
49	1.671080	0.074691	0.033222	0.001536	10	200	3.0	10	0.40	0.40	0.310511	0.301344	0.301850	0.289970	0.317121	0.304159	0.009197	8
12	0.556033	0.016523	0.023616	0.001066	10	50	1.5	5	0.90	0.00	0.312343	0.305506	0.284960	0.297297	0.303556	0.300732	0.009232	9
45	1.024397	0.105854	0.038746	0.002976	5	150	3.0	4	1.00	0.40	0.301035	0.300917	0.280702	0.309413	0.285171	0.295448	0.010763	10

'XGBoostPRAUCBase:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
41	4.712076	0.102368	0.083338	0.028096	3	250	3.0	7	0.01	0.10	0.788226	0.775678	0.771499	0.759392	0.771165	0.773192	0.009269	1
27	2.457221	0.134930	0.042669	0.002389	10	150	2.0	10	0.20	0.10	0.773451	0.762996	0.781222	0.761001	0.768260	0.769386	0.007335	2
8	1.363308	0.106730	0.038190	0.001965	3	150	2.5	5	0.05	0.05	0.781327	0.771987	0.769861	0.763578	0.759664	0.769283	0.007453	3
23	3.857984	0.076118	0.052538	0.008277	3	250	1.5	12	0.10	0.30	0.774194	0.780919	0.772242	0.754127	0.758109	0.767918	0.010134	4
42	2.214649	0.101345	0.052262	0.008496	10	200	2.0	7	0.20	0.10	0.777103	0.772021	0.773124	0.760034	0.755120	0.767480	0.008406	5
65	1.996158	0.045136	0.044282	0.001806	3	150	1.5	6	0.05	0.10	0.774194	0.767031	0.771574	0.765289	0.757417	0.767101	0.005786	6
62	5.784741	0.083577	0.077049	0.013330	3	200	1.0	10	0.05	0.40	0.771993	0.771041	0.778571	0.757840	0.751148	0.766119	0.010064	7
61	1.500142	0.040802	0.036259	0.001377	3	200	3.0	7	0.20	0.30	0.775439	0.776408	0.763766	0.750647	0.763276	0.765907	0.009439	8
40	1.545100	0.082604	0.028552	0.000674	3	250	2.0	12	0.40	0.10	0.786738	0.765077	0.756757	0.749350	0.771611	0.765906	0.012846	9
47	0.764223	0.023834	0.025007	0.000622	2	50	2.5	7	0.20	0.40	0.783760	0.772208	0.767442	0.738488	0.754579	0.763295	0.015536	10

'XGBoostTunePRAUC:'

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
46	1.167071	0.107811	0.034444	0.015000	5	50	3.0	7	0.10	0.00	0.843684	0.835045	0.835531	0.809094	0.820630	0.828797	0.012336	1
45	5.331813	0.371688	0.085230	0.024038	1	200	2.5	10	0.05	0.05	0.835988	0.838556	0.830562	0.805956	0.823805	0.826973	0.011662	2
35	7.442352	0.189858	0.072972	0.003792	2	200	3.0	12	0.01	0.30	0.847388	0.829016	0.832578	0.797764	0.819548	0.825259	0.016407	3
1	4.444903	0.116943	0.057768	0.002373	5	150	2.5	10	0.05	0.40	0.832729	0.834509	0.831418	0.801665	0.822907	0.824646	0.012162	4
39	0.572882	0.066626	0.027122	0.007576	5	50	3.0	5	0.20	0.00	0.837891	0.834781	0.822234	0.808294	0.815544	0.823749	0.011227	5
27	2.394500	0.106129	0.053726	0.004209	2	250	2.0	7	0.10	0.40	0.828421	0.832822	0.827069	0.805578	0.818790	0.822536	0.009619	6
38	0.901095	0.013271	0.031447	0.001048	3	150	2.0	4	0.10	0.10	0.831417	0.832216	0.822399	0.802439	0.814818	0.820658	0.011121	7
55	2.220437	0.304225	0.034987	0.004498	10	250	3.0	12	0.20	0.40	0.828787	0.832201	0.820685	0.801097	0.815254	0.819605	0.011006	8
51	6.164659	0.160574	0.085509	0.020234	2	200	1.0	10	0.05	0.40	0.832165	0.825742	0.823084	0.797151	0.818171	0.819262	0.011943	9
24	7.032639	0.165788	0.067873	0.002149	3	150	1.0	12	0.05	0.00	0.838111	0.823868	0.823744	0.798606	0.806830	0.818232	0.013946	10

Dropping cols where: grade is missing 226070 -> 226067

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

	best_score	best_params
model_key
XGBoostOrdinalGradeRegressor_Default	-1.304155	{'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostMulticlassTunePRAUC	0.407976	{'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 0.5, 'model__max_depth': 12, 'model__learning_rate': 0.2, 'model__gamma': 0}
XGBoostMulticlassTuneLogLoss	-1.287763	{'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 100, 'model__min_child_weight': 2.5, 'model__max_depth': 4, 'model__learning_rate': 0.05, 'model__gamma': 0}
XGBoostF1Multiclass	0.439000	{'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.3, 'model__gamma': 0.1}

'\n Model: XGBoostOrdinalGradeRegressor_Default'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	-1.248444
169	1	1	1	0	0	0	1	-1.248444
168	1	1	1	0	0	0	0	-1.248444
127	1	0	1	0	1	0	1	-1.248444
126	1	0	1	0	1	0	0	-1.248444
121	1	0	1	0	0	0	1	-1.248444
120	1	0	1	0	0	0	0	-1.248444
174	1	1	1	0	1	0	0	-1.248444
78	0	1	1	0	1	0	0	-1.248444
73	0	1	1	0	0	0	1	-1.248444

'Transformer impact for XGBoostOrdinalGradeRegressor_Default'

	Transformer_Option	Impact
4	feat_trans_inst_income_ratio__option_1	0.0176
5	feat_trans_new_dti_after_loan__option_1	0.0082
3	feat_trans_fico_score__option_1	0.0000
1	feat_trans_dti_inc_joint__option_1	-0.0011
0	feat_trans_delinquency__option_1	-0.0022
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.3815

XGBoostOrdinalGradeRegressor_Default

'\n Model: XGBoostMulticlassTunePRAUC'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	0.244104
169	1	1	1	0	0	0	1	0.244104
168	1	1	1	0	0	0	0	0.244104
127	1	0	1	0	1	0	1	0.244104
126	1	0	1	0	1	0	0	0.244104
121	1	0	1	0	0	0	1	0.244104
120	1	0	1	0	0	0	0	0.244104
174	1	1	1	0	1	0	0	0.244104
78	0	1	1	0	1	0	0	0.244104
73	0	1	1	0	0	0	1	0.244104

'Transformer impact for XGBoostMulticlassTunePRAUC'

	Transformer_Option	Impact
0	feat_trans_delinquency__option_1	0.0019
4	feat_trans_inst_income_ratio__option_1	0.0016
3	feat_trans_fico_score__option_1	0.0000
1	feat_trans_dti_inc_joint__option_1	-0.0019
5	feat_trans_new_dti_after_loan__option_1	-0.0049
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.1474

XGBoostMulticlassTunePRAUC

'\n Model: XGBoostMulticlassTuneLogLoss'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	-1.485756
169	1	1	1	0	0	0	1	-1.485756
168	1	1	1	0	0	0	0	-1.485756
127	1	0	1	0	1	0	1	-1.485756
126	1	0	1	0	1	0	0	-1.485756
121	1	0	1	0	0	0	1	-1.485756
120	1	0	1	0	0	0	0	-1.485756
174	1	1	1	0	1	0	0	-1.485756
78	0	1	1	0	1	0	0	-1.485756
73	0	1	1	0	0	0	1	-1.485756

'Transformer impact for XGBoostMulticlassTuneLogLoss'

	Transformer_Option	Impact
4	feat_trans_inst_income_ratio__option_1	0.0002
1	feat_trans_dti_inc_joint__option_1	0.0000
3	feat_trans_fico_score__option_1	0.0000
0	feat_trans_delinquency__option_1	-0.0003
5	feat_trans_new_dti_after_loan__option_1	-0.0033
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.1490

XGBoostMulticlassTuneLogLoss

'\n Model: XGBoostF1Multiclass'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	0.33
169	1	1	1	0	0	0	1	0.33
168	1	1	1	0	0	0	0	0.33
127	1	0	1	0	1	0	1	0.33
126	1	0	1	0	1	0	0	0.33
121	1	0	1	0	0	0	1	0.33
120	1	0	1	0	0	0	0	0.33
174	1	1	1	0	1	0	0	0.33
78	0	1	1	0	1	0	0	0.33
73	0	1	1	0	0	0	1	0.33

'Transformer impact for XGBoostF1Multiclass'

	Transformer_Option	Impact
4	feat_trans_inst_income_ratio__option_1	0.0012
3	feat_trans_fico_score__option_1	0.0000
0	feat_trans_delinquency__option_1	-0.0003
5	feat_trans_new_dti_after_loan__option_1	-0.0035
1	feat_trans_dti_inc_joint__option_1	-0.0067
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.1040

XGBoostF1Multiclass